Adding kv-cache invalidation when generation hits short factor threshold by AlexanderKalistratov · Pull Request #4117 · openvinotoolkit/openvino.genai

AlexanderKalistratov · 2026-07-08T18:29:18Z

Description

Add support for short-factor to long-factor RoPe transition via cache invalidation for SDPA backend.
This is a functional support of the feature as cache-invalidation has performance impact.
Tested on CPU and NPU plugins.
Currently this functionality is enabled for NPU only, but can be extended to other plugins.

EISW-223126

Checklist:

This PR follows GenAI Contributing guidelines.
Tests have been updated or added to cover the new code.
This PR fully addresses the ticket.
I have made corresponding changes to the documentation.

Copilot

Pull request overview

This PR adds LongRoPE “short-factor → long-factor” transition handling for the stateful LLM pipeline by detecting the model’s LongRoPE switch threshold from config.json and forcing KV-cache invalidation / reprefill when crossing that boundary (including a mid-generation split for SDPA).

Changes:

Detect LongRoPE usage + threshold from config.json and store it in StatefulLLMPipeline.
Force a one-off full-history resend + cache reset when a chat turn crosses the threshold; add a streamer-based mid-generation stop-and-rerun path.
Ensure LongRoPE threshold detection is invoked in stateful pipeline construction paths.

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 4 comments.

File	Description
`src/cpp/src/lm_encoding.cpp`	Minor formatting/whitespace updates.
`src/cpp/src/llm/pipeline.cpp`	Calls `StatefulLLMPipeline::set_longrope_threshold(models_path)` after construction to initialize LongRoPE behavior.
`src/cpp/src/llm/pipeline_stateful.hpp`	Adds LongRoPE threshold state and “one-shot” reprefill flag declarations.
`src/cpp/src/llm/pipeline_stateful.cpp`	Implements LongRoPE threshold detection and adds cache-reset + rerun logic when the boundary is reached.

Copilot

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 5 comments.

Copilot

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.

…r & debug output

Copilot

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 1 comment.

Adding kv-cache invalidation when generation hits short factor threshold

896ea55

AlexanderKalistratov requested a review from pavel-esir as a code owner July 8, 2026 18:29

Copilot AI review requested due to automatic review settings July 8, 2026 18:29

AlexanderKalistratov requested review from Wovchena and as-suvorov as code owners July 8, 2026 18:29

AlexanderKalistratov added this to the 2026.3 milestone Jul 8, 2026

github-actions Bot added category: visual language Visual language pipeline category: LLM LLM pipeline (stateful, static) labels Jul 8, 2026

Copilot started reviewing on behalf of AlexanderKalistratov July 8, 2026 18:29 View session

Copilot AI reviewed Jul 8, 2026

View reviewed changes

Comment thread src/cpp/src/llm/pipeline_stateful.cpp

Comment thread src/cpp/src/llm/pipeline_stateful.cpp

Comment thread src/cpp/src/llm/pipeline_stateful.cpp

Comment thread src/cpp/src/llm/pipeline_stateful.cpp

AsyaPronina approved these changes Jul 9, 2026

View reviewed changes

Comment thread src/cpp/src/llm/pipeline_stateful.cpp Outdated

AsyaPronina reviewed Jul 9, 2026

View reviewed changes

Comment thread src/cpp/src/llm/pipeline_stateful.cpp Outdated

Addressing review comments & make it NPU only

a7879eb

Copilot AI review requested due to automatic review settings July 9, 2026 14:36

Copilot started reviewing on behalf of AlexanderKalistratov July 9, 2026 14:36 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Comment thread src/cpp/src/llm/pipeline_stateful.cpp

Comment thread src/cpp/src/llm/pipeline_stateful.cpp Outdated

Comment thread src/cpp/src/llm/pipeline_stateful.cpp

Comment thread src/cpp/src/llm/pipeline_stateful.hpp

Comment thread src/cpp/src/llm/pipeline_stateful.cpp

AlexanderKalistratov added 2 commits July 9, 2026 21:57

Fix forwarding end twice & handle corener case

835b0d9

Return correct finish status on edge case

72059de

Copilot AI review requested due to automatic review settings July 9, 2026 22:39

Copilot started reviewing on behalf of AlexanderKalistratov July 9, 2026 22:40 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Comment thread src/cpp/src/llm/pipeline_stateful.cpp

Comment thread src/cpp/src/llm/pipeline_stateful.cpp

Copilot AI review requested due to automatic review settings July 10, 2026 00:58

Copilot started reviewing on behalf of AlexanderKalistratov July 10, 2026 00:59 View session

Fix corner case when treshold is reached but stopped but base streame…

726a031

…r & debug output

AlexanderKalistratov force-pushed the longrope_sdpa_kvcache_invalidation branch from 4360de1 to 726a031 Compare July 10, 2026 01:01

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread src/cpp/src/llm/pipeline_stateful.cpp

Comment thread src/cpp/src/llm/pipeline_stateful.cpp Outdated

Comment thread src/cpp/src/llm/pipeline_stateful.hpp

Copilot AI review requested due to automatic review settings July 10, 2026 01:02

Copilot started reviewing on behalf of AlexanderKalistratov July 10, 2026 01:02 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread src/cpp/src/llm/pipeline_stateful.cpp

Comment thread src/cpp/src/llm/pipeline_stateful.cpp Outdated

more debug output

d140006

Copilot AI review requested due to automatic review settings July 10, 2026 01:38

Copilot started reviewing on behalf of AlexanderKalistratov July 10, 2026 01:38 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread src/cpp/src/lm_encoding.cpp Outdated

Comment thread src/cpp/src/llm/pipeline_stateful.cpp

Comment thread src/cpp/src/llm/pipeline_stateful.cpp Outdated

Adjust max_kv_cache_size, remove debug output and adress review comments

75d33b6

Copilot AI review requested due to automatic review settings July 10, 2026 02:41

Copilot started reviewing on behalf of AlexanderKalistratov July 10, 2026 02:42 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread src/cpp/src/llm/pipeline_stateful.cpp

Uh oh!

Conversation

AlexanderKalistratov commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AlexanderKalistratov commented Jul 8, 2026 •

edited

Loading